Harvester with automated targeting capabilities

ABSTRACT

Systems and methods here may include a vehicle with automated subcomponents for harvesting delicate items such as berries. In some examples, the vehicle includes a targeting subcomponent and a harvesting subcomponent. In some examples, the targeting subcomponent utilizes multiple cameras to create three-dimensional maps of foliage and targets. In some examples, identifying targets may be done remotely from the harvesting machine, and target coordinates communicated to the harvesting machine for robotic harvesting.

CROSS REFERENCE

This application relates to and claims priority to U.S. Provisionalapplication 62/796,319 filed Jan. 24, 2019 the entirety of which ishereby incorporated by reference.

TECHNICAL FIELD

This application relates to the field of automated agriculturalharvesting equipment and methods using robotic assemblies, mobileharvesting units, remote harvesting, sensors for target identification,target tracking, and various combinations of related technologies.

BACKGROUND

The agriculture industry is highly reliant on human pickers to harvest anumber of produce, including berries such as strawberries. The reasonhuman pickers are still used today, despite the technologicaladvancements available, is because of the difficulty of identifying atarget such as a berry in a field, that is ready to be picked, reachingthrough the foliage of the plant to grasp that berry, and then carefullyremoving that berry without damaging it, to package and sellimmediately.

Current automatic harvesting of such delicate and difficult to graspagricultural targets such as berries, while operating in a harsh outdoorenvironment did not exist before this application.

SUMMARY

Systems and methods here may include a vehicle having varioussubcomponents for harvesting delicate agricultural items such asberries. In some examples, the subcomponents may be automated. In someexamples, the vehicle may include a targeting subcomponent and aharvesting subcomponent. In some examples, the targeting subcomponentutilizes multiple cameras to create three-dimensional maps of the targetand target areas sometimes including the agricultural foliage. In someexamples, the targeting subcomponent may include any of various cameras,sensors, or other targeting features to locate and map targets in anautomated or semi-automated manner. The system may then determinecoordinates of the mapped targets to be passed to the harvestingsubcomponent. In some examples, the harvesting subcomponent may includevacuum features which help a nozzle attach to an agriculture target forharvesting. In some examples, the harvesting subcomponent includespadded spoons to aid in removal of the targeted agriculture from theplant, including in some examples, a stem.

Systems and methods here include a harvesting vehicle system including aharvesting vehicle frame with computing device includes at least oneprocessor and a memory including picking control systems, navigationsystems, and communication systems, a picking subcomponent including arobotic arm and servo camera in communication with the computing device,additionally or alternatively, the robotic arm including a picker headassembly to harvest targets including a vacuum assembly with acompressor, hose, and padded spoons configured to remove the target froma target stem, additionally or alternatively, vehicle including multiplecameras in communication with the computing device, wherein the camerasare configured to capture and send image data to the computing device,additionally or alternatively, the computing device further configuredto create three-dimensional maps of targets using the multiple cameraimage data, additionally or alternatively, the computing device furtherconfigured to direct the robotic arm and picker head to a target toharvest using the mapped coordinates, the picker head assemblyconfigured to attach the vacuum assembly and padded spoons to the mappedtarget, and retract the target for harvesting. In some examples,additionally or alternatively, the computing device is configured tosend the image data to a computer over a network and receive targetselection from the image data. In some examples, additionally oralternatively, the received target selection regarding the image datafrom the network includes a selection of a category of each selectedtarget. In some examples, additionally or alternatively, the categoriesof each selected target include grade, spoiled, immature, or ready topick. In some examples, additionally or alternatively, the selectedtargets are selected by the computing system, using imbedded neuronetwork logic, trained from models of human selected targets classifiedas ready-to-pick, immature, or spoiled. In some examples, additionallyor alternatively, the computer is further configured to utilize close insensors to direct the picker head to a target once the picker head iswithin a predetermined distance from the target using thethree-dimensional map. In some examples, additionally or alternatively,the communication system includes wireless communication devices, incommunication with the computing device, configured to send and receivedata regarding navigation and camera image data to wireless antenna incommunication with a back-end computing system. In some examples,additionally or alternatively, the navigation systems include at leastone of Global Positioning System, Inertial Measurement systems,Simultaneous Localization And Mapping systems, and an Odometer. In someexamples, additionally or alternatively including a back-end computingsystem configured to cause display interface of the camera image datafor a user, allow touch screen selection of targets and determination ofcoordinates for the selected targets to be sent to the harvestingvehicle computing device for picking by the picker head. In someexamples, additionally or alternatively, the coordinates of the selectedtarget are sent to a queue buffer at the harvesting vehicle computingdevice for picking by the picker head in queue order. In some examples,additionally or alternatively, the display of the camera image dataincludes preselected targets, preselected by the back-end computingsystem, based on training of models of targets, wherein the displayinterface allows users to affirm or change the preselected targets forharvesting.

Systems and methods of harvesting agriculture described herein includetraversing a harvesting vehicle frame across a row of agriculturalplants wherein the harvesting vehicle includes a computing device with aprocessor and a memory, wherein the computing device including targetacquisition control, picking control, wherein the harvesting vehicleincluding a picking subcomponent with a robotic arm with a picker headassembly, wherein the robotic arm in communication with the computingdevice, the picker head assembly including a vacuum assembly with acompressor, hose, and padded spoons, capturing and sending image data tothe target acquisition control of the computing device, using multiplecameras on the harvesting vehicle, identifying targets in theagricultural plants, by the target acquisition control of the computingdevice, using the image data, creating three-dimensional maps oftargets, by the computing device using the image data, directing, by thepicker control of the computing device, the robotic arm and picker headto a selected target using the three-dimensional maps of targets, andharvesting, by the picker control of the computing device, the targetwith the picker head assembly by attaching the vacuum assembly andpadded spoons to the mapped target, and retracting the target. In someexamples, alternatively or additionally, the traversing and navigationof the harvesting machine is controlled by navigation control in thecomputing device. In some examples, alternatively or additionally,sending and receiving target acquisition and navigation data from acommunication control in the computing device with an off-board system.In some examples, alternatively or additionally, the communicationcontrol includes communicating using wireless communication devices bysending and receiving data regarding navigation and camera image data bywireless antenna in communication with a back-end computing system. Insome examples, alternatively or additionally, harvesting with the pickerhead assembly includes sending commands to a picker head actuator topinch padded spoons together to secure a target, the target beingidentified by the computing device target acquisition control. In someexamples, alternatively or additionally, harvesting includes receivingdata at the computing device target acquisition control, from close insensors on the harvesting machine, directing the picker head to anidentified target, by the computing device target acquisition controlonce the picker head is within a predetermined distance from the target,determined using the three-dimensional maps. In some examples,alternatively or additionally, the target acquisition data from theoff-board system includes target selection with selection of a categoryof each selected target. In some examples, alternatively oradditionally, the systems and methods may include causing display, withthe off-board system, of a display interface of the camera image datafor a user, and allowing touch screen selection of targets to be sent tothe harvesting vehicle computing device for picking by the picker head.

BRIEF DESCRIPTION OF THE DRAWINGS

For a fuller understanding of the nature and objects of the disclosure,reference should be made to the following detailed description, taken inconnection with the accompanying drawings, in which:

FIGS. 1A and 1B are diagrams showing example mobile vehicle examples asdescribed in the embodiments disclosed herein.

FIG. 2 is a diagram showing example picker head example details asdescribed in the embodiments disclosed herein.

FIG. 3 are diagrams showing example conveyor belt examples as describedin the embodiments disclosed herein.

FIG. 4 are diagrams showing example sensor examples as described in theembodiments disclosed herein.

FIG. 5 is an example stereoscopic camera arrangement that may be used inthe embodiments disclosed herein.

FIG. 6 is an example GUI may be used in the embodiments disclosedherein.

FIG. 7 is another example GUI may be used in the embodiments disclosedherein.

FIG. 8 is an illustration of an example networked system in accordancewith certain aspects described herein; and

FIG. 9 is an example computer architecture arrangement that may be usedin the embodiments disclosed herein.

FIG. 10 is an example computer architecture arrangement that may be usedin the embodiments disclosed herein.

FIG. 11 is an example computing system which may be used in theembodiments disclosed herein.

DETAILED DESCRIPTION

Reference will now be made in detail to embodiments, examples of whichare illustrated in the accompanying drawings. In the following detaileddescription, numerous specific details are set forth in order to providea sufficient understanding of the subject matter presented herein. Butit will be apparent to one of ordinary skill in the art that the subjectmatter may be practiced without these specific details. Moreover, theparticular embodiments described herein are provided by way of exampleand should not be used to limit the scope of the invention to theseparticular embodiments. In other instances, well-known data structures,timing protocols, software operations, procedures, and components havenot been described in detail so as not to unnecessarily obscure aspectsof the embodiments of the invention.

Overview

The systems and methods described here include an automated and/orsemi-automated system with machine(s) that is/are capable of harvestingagricultural targets such as berries from their planter beds withouthuman hands touching the plants or targets themselves. This presentsdramatic improvements in efficiency, productivity, more sanitary, andless strenuous agricultural harvesting than has ever been accomplished.These achievements may be made in the field, where agricultural plantsare best suited and easily grown, yet harvested using machines, computermethods, remote target selection and/or combinations of these and othertechnologies.

Example overall systems may include subcomponents such as a seeker orsensor subsystem to find and locate the targets that works with andinforms a robotic picking subsystem to harvest the targets. The overallsystem(s) may be mounted on wheels or tracks to advance down a row oftargets such as agricultural produce planter bed rows so that the seekersubassembly may identify and map the targets while the picker subsystemis used to harvest, gather, pack and/or move the targets.

In some example embodiments, additionally or alternatively, the seekersubassembly includes a camera or multi-camera system such as astereoscopic arrangement to capture image data to be sent to awirelessly connected remote operator(s) to locate target berries andthree-dimensionally map them. In such examples, these mapped targetcoordinates may then be queued for harvesting using the roboticassemblies. Additionally or alternatively, in some examples, theharvester subassembly is then able to follow the seeker subassembly andharvest the target berries whose mapped locations are queued by theseeker subassembly. In some example embodiments, the harvestersubassembly includes at least one robotic arm with multiple degrees offreedom capable of reaching into the foliage of a plant and extracting atarget such as a berry. In some examples, additionally or alternativelythe extraction is aided by a vacuum system. In some examples,additionally or alternatively, the extractions are aided by a paddedspoon grasper, capable of grasping, and in some examples, twisting andsnapping a berry stem.

It should be noted that the examples used here describing berryharvesting, or even berry or strawberry harvesting in the writtendescription and/or figures is not intended to be limiting and are merelyused as examples. The agricultural targets to which the systems here mayidentify, map, and ultimately harvest may be any sort including but notlimited to berries such as strawberries, blackberries, blueberries, andraspberries, other examples include grapes, figs, kiwi, dragon fruit, orother fruits. Vegetables may be harvested as well, such as Brusselsprouts, tomatoes, peppers, beans, peas, broccoli, cauliflower, or othervegetable. Any type of agricultural target may be harvested using thesystems described herein. Additionally or alternatively, the systems andmethods here may be used to target and gather non-agricultural itemssuch as garbage, or be used to take scientific samples such as rocks orminerals in environments or situations where it may be advantageous toavoid human contact or interaction.

Harvester Subassembly

In some example embodiments, a harvesting subassembly is included as itsown separate vehicle system from the seeker subassembly. Additionally oralternatively, in some examples, the harvesting subassembly may be incommunication with or connected to the seeker subassembly. In someexamples, seeker subcomponents are integrated into the harvestingassembly and one overall frame/chassis of a machine incorporates all ofthe features described herein. The harvesting subassembly may includeany number of features that allow for autonomous, semi-autonomous,and/or manual human operable harvesting of delicate target agriculturesuch as berries, as described herein.

In such examples, computer components may be in communication with thevarious sensors and robotic arms and picker assemblies to locate,identify, and pick agricultural targets. In some examples, wirelesscommunications may send and receive data from the harvester and sensorsand picker arms to a remote computer which may include a graphical userinterface (GUI) that displays the sensor data from the sensors, to allowa computer and/or human to identify targets to pick, thereby allowingthe picker assemblies to utilize coordinate data from the sensors topick targets. More detailed discussion of such features is found inFIGS. 6, 7, 5, 8, 9, 10, and 11 and accompanying descriptions below.

Regarding the overall picking machine, in some examples, eitherharvesting or seeker subassemblies may be mounted on its own vehiclesubassembly with wheels and/or tracks or combination of both, totraverse down a row of agriculture with the seeker subassemblyidentifying and mapping the target berries and the harvester subassemblygathering targets.

FIGS. 1A and 1B show different view of an example of the overalltraversing machine to which any of the various subassemblies may beattached and/or coupled. In the example, the main traversing subassembly152 includes various portions mounted to it including main drivingwheels 154 and in some examples, guide wheels 156. In some examples,guide wheels 156 may be canted outward in order to support traversing amound 101 should a mound be configured. In some examples, tank treads ortracks may be used instead of wheels 154, and/or a combination of wheelsand tracks may be used.

In some examples, a robotic arm 160 or arms may be mounted to any ofvarious frame portions 153 and/or chassis portions 155 that comprise theoverall traversing subassembly 152. In some examples, the robotic arm160 may include at least one picker head, at least one sensor, at leastone light system, and/or a combination of picker heads, sensors, and/orlights to locate, identify, and harvest agricultural targets such asberries.

For example purposes, the range 195 of the robotic arm 160 is shown inthe FIGS. 1A and 1B to show that the robotic arm 160 may reach differentsides of the planter bed or row mound 101 where targets may be found andan accumulator for processing targets, such as the example traversingconveyor 178.

In some example embodiments, the harvesting subassembly may include atleast one robotic arm 160 with joints that allow for multiple degrees offreedom. Such arms 160 may be configured to maneuver around and infoliage 180 of a target plant to extract the target agriculture such asbut not limited to berries of any sort. In example embodiments, therobotic arm 160 may include various numbers of joints thereby allowingfor various degrees of freedom to move around and about the plants androws, taking different angles. In some examples, the robotic arm 160 mayinclude six degrees of freedom. In some examples, the robotic arm 160may include seven degrees of freedom, or any other number. In variousexample embodiments, the robotic arm 160 may be any of various lengths,thereby affecting the range 195 of the arm, which may be tailored to theneeds of the particular field or mound or target. In some examples, therobotic arm 160 may include one or more telescoping portions, which maybe elongated and/or retracted, thereby affecting the length of thatportion and the overall reach 195 of the robotic arm 160.

It should be noted that many variations of robotic arms 160 may be usedin the systems described here, including but not limited to roboticwrists with link and joint combinations with linear and rotationallinks, gantry robots with linear joints, cylindrical robots connected torotary base joints, polar robots for twisting, and/or jointed-arm orarticulating robots with twisting joints and rotary joints. Anycombination of these or other robotic assemblies 160 may be used on thesystems described herein to manipulate a picker head and/or sensors forharvesting agricultural targets as described.

It should be noted that the system in FIG. 1A, 1B is shown straddlingone row of plants. In some examples, one system may straddle two, three,or wider mounds of plants and the example in FIG. 1A, 1B are merelyintended to be an example, and not limiting. By making the system widerto straddle a second row, two sets of arms 160 may be used to pick tworows, or three, or four, or whichever number.

In some examples, multiple robotic arms 160 may be fit onto one overalltraversing vehicle 152. For example, systems may include a primarypicker assembly with a clean-up/redundant picker assembly which operatesbehind the primary setup. In those examples, up to eight picker arms 160may be employed, four on the primary and four on the clean-up assembly,with one or two arms operating on each side of two rows. The clean-upsystem may operate in the same way that the primary system operates, tofind targets that the primary system did not harvest, and/or to operateas a redundancy should one or more arms on the primary systemmalfunction.

In examples where targets are fruit plants which are harvested manymultiple times during a single growing season, often multiple times perweek, leaving fruit on a fruit plant may curtail the productivity of theplant. If the plant senses that it still has fruit on it, it may notproduce more fruit. This would limit production, so the goal is toremove all of the fruit when ripe. As the bed rows may allow for somefruit to drape over the side of the plastic wrapped bed rows 101, andbecome easily exposed to viewing, other fruit may grow under the foliage180, or on top of the bed row crowns and be obscured by foliage 180.Therefore, to find and harvest as much fruit from each plant aspossible, it may be necessary to maneuver the foliage 180 to better viewand/or harvest fruit targets as described herein.

In some examples, robotic arms 160 may include foliage moving featuresto alter, move, displace, and/or otherwise gently maneuver the foliage180 of the plant to better expose the targets such as fruit berries tobe picked. In such examples, a bar, or arm, may be pulled across the topof the foliage 180 in order to temporarily move it out of the way forthe seeker cameras and/or the harvesting assembly to locate and grapplethe target. In some examples, this foliage moving arm 160 may bemaneuvered parallel or substantially parallel to the top of the row bed101, and pull across the top of the foliage 180, bending the plant, butnot breaking the plant leaves. This may reveal targets under the foliage180, those laying on the top of the row bed 101, or those caught up inthe foliage 180.

In some examples, a flexible curtain (not shown) may be dragged over thefoliage 180, to avoid damage to the foliage, but still pull it out ofthe way for the seeker and/or harvester to operate. In some examples,this flexible curtain may be a plastic skirt, in some examples, it maybe a fringed or sliced skirt. In some examples, it may have fringes thatdrape over the foliage 180, and yet flex around the foliage 180 so asnot to damage it. As the flexible skirt is pulled over the plants 180,it may thereby help the seeker subassembly find the targets more easilyby limiting the area to be targeted with a clean backdrop. The flexibleskirt may be dragged from one side in one direction during a firstharvest and the next time the other direction, to avoid biasing orpulling the foliage in the same direction each time.

In some examples, the overall traversing subassembly 152 may include atransfer conveyor 178. Such a conveyor may include any number ofconveyor belts, chains, rope, or other mechanism that can pull materialsfrom one place to another. Such transfer conveyor may be used to collectharvested targets from the robotic arm 160 which picks the targets fromthe plants 180 and moves them to a packaging subassembly, or storageunit.

In some example embodiments, the robotic arms 160 may be ruggedized inthat the tolerances and durability of the arms are developed foroutside, dirty employment. In such examples, the robotic arms are not tobe operated in clean, pristine factory settings. The systems describedhere will operate in weather, precipitation, dirt, mud, heat, cold, andin jarring, rough conditions. As such, the bearings, tolerances, andactuators may be made of more durable materials than factory roboticassemblies. In some examples, extra gaskets may be fitted into thevarious robotic arm joints to keep dirt out of the more delicate metalcouplings and pivoting features of the robotic arms 160. In suchexamples, gaskets may be made of rubber, plastic, and/or ceramic. Therobotic arms 160 may be made with fewer joints to minimize the number ofpotential problems that may occur. The robotic arms 160 may be made ofthicker materials, may be heavier, and be rust-proofed, waterproof,weatherized, and/or otherwise reinforced.

Picker Head Examples—Vacuum Point of Contact

In some example embodiments, the harvesting subassembly may include atleast one picker head at the end of the robotic arm 160 that firstinteracts with the target in the field to remove or detach the targetfrom the plant 180 it grows on. Such picker heads may be affixed to orbe part of the robotic arms 160 as discussed in FIG. 1A, 1B. In someexample embodiments, the at least one picker head may be mounted on orpartially mounted on a robotic harvesting arm 160, alone or incombination with sensors such as cameras and/or lighting system(s).

FIG. 2 shows an example picker head assembly which may be mounted to arobotic arm, with a front 2A and side 2B view of the same assembly indetail. The picker head assembly is designed to grasp and remove targetsfrom the plants. Computer systems may be in communication with thecomponent parts of the picker head assembly to operate it, as describedherein. Many various features may be utilized alone or in combination toaccomplish this task, and the example of FIG. 2 is one example, notintended to be limiting.

In the example shown in FIG. 2, the main picker head assembly 202 may bemounted with two actuators 204, 206, which are shown as pneumatic orfluid filled pistons that may extend or retract when actuated. In suchexamples, a computer system which may be in communication with therobotic assembly, sensors, lighting, etc. may also command the actuatorsto extend and/or retract, thereby actuating their respective componentpart.

In the example, one actuator may be utilized for the pinching, grapplingspoons 204 and one actuator for an extender 206. In such examples, theactuator for the extender 206 may simply be attached to a bracketholding the nozzle 203, thereby extending or retracting the nozzle 203with the actuator 206. In some examples, the actuator for the grapplingspoons 204, may be in communication with a bracket 216 that when movedup or down, may contact the ends of the grappling spoons 270, 272, onone side of pivot or axis points on a fixed bracket 218, thereby openingand closing the spoon arms 212, 214 as the bracket 216 is slid up anddown the ends of the spoon arms 270, 272 due to the tapered nature ofthe ends of the spoon arms 27, 272.

In some examples, no extender actuator 206 may be utilized, and thevacuum tube 203 may remain stationary in relation to the spoons 212,214. In examples where an extender 206 is utilized, the extenderactuator 206 may move in and down to move the main nozzle 203 up anddown, relative to the robotic arm (as shown in FIG. 3).

In some examples, the picker head assembly may utilize a vacuum featureto help grasp the target. In such examples, a main nozzle 203 may be ahollow tube in communication with a vacuum pump (not shown) which maylower air pressure in the tube 203 to be used to secure a couplingsuction portion 230 to a target 250 such as a berry. In some examples,the coupling portion 230 may include one or more bellows or bellowconfigurations that allow the coupling portion to stay flexible andmalleable to couple with the target 250. In some examples, the vacuumhose 203 may be connected with the main nozzle 230 to impart a suctionor lower than ambient pressure within the nozzle tube 203, and therebybe able to attach to and secure a target 250. In some examples, a vacuumsubsystem with a pump may be mounted on the harvesting subassembly and avacuum hose may run through or around each harvesting picker roboticarm. In some examples, vacuum subassemblies may be mounted on therobotic arm itself, along with a vacuum hose on the picker head 202.

In some examples, the compression coupling portion 230 may be 1.250inches in diameter, in some examples, the nozzle may be between 1.000and 0.750 inches in diameter, but in any case, the nozzle could becustomized to any size of intended target.

In some examples, the amount of suction power that the vacuum systemimparts, may be 35 inches of vacuum. In some examples, 50 inches ofwater-column vacuum may be used. Alternatively or additionally, in someexamples, less than 80 inches water-column of vacuum may be used so asto avoid damage to the target 250. In some examples, less than 65 incheswater-column of vacuum may be used to avoid damaging targets 250.Alternatively or additionally, in some examples, the amount of suctionpower may be between 35-50 inches water-column of vacuum may be used.Additionally or alternatively, the vacuum system may be able to reversefrom suction to blowing air outward, to clear debris, before switchingback to a suction mode for harvesting.

The compression nozzle portion 230 may include a malleable hood orcoupling section 232 which may include one or more bellow sections, anda rim 234 around an opening 236 to aid in coupling to a target. In someexamples, the compression coupling portion 230 is or made up of at leastone of, or combination of a neoprene sleeve, a silicone sleeve, a rubbersleeve, or other natural or synthetic material that is soft andflexible. Such a malleable coupling section 232 may be configured todeform or otherwise compress when a target 250 is contacted and mayinclude baffles or other structure that allows for deformation andmalleability. Such a deformation or compression may allow for the rim234 to more easily conform to the target 250 and thereby form a bettersuction fit for the opening 236 on the target 250.

In some examples, the compression nozzle portion 230 may include aninternal reverse conical mesh at the end 236 to help capture the target250 yet be as gentle as possible on them. In such examples, the mesh maycreate an environment where the vacuum is acting on a broader surface ofthe target, thus minimizing the chance of target damage from localizedcontact to the grappler edges. This mesh 236 may thereby form a cradlefor the target to lay in even while being picked and moved. Such a mesh236 can be made of silicone materials for durability and flexibility.Alternate materials may be used such as a wire mesh, a plastic mesh, ora combination of wire mesh with plastic coating. Silicon coating may beused on a wire mesh in some example embodiments as well.

In some examples, the compression nozzle portion 230 and the opening rim234 may be sized for a most average target 250, big enough for thebiggest targets and flexible, but able to grasp and vacuum even asmaller target.

Examples may also include an internal spring system, inside orintegrated into the coupling portion 230. Such a spring system may bemade of plastic or metal coil(s) that help return the coupling portion230 back to an extended shape after a target is released by turning offthe vacuum and thereby deposited the target 250. Additionally oralternately, a mechanical iris or camera lens type feature may beintegrated into the nozzle 230 and in communication with the computersystems. In such examples, the system may be able to adjust the size ofthe opening or nozzle 203 end for different sized targets 250, openingfor larger targets, and constricting for smaller targets to control thesize of the opening and thereby the amount of air being vacuumed. Insuch examples, a coil or spring could be wound tighter for smallertargets and wound looser for larger targets.

Another portion of the example embodiment of FIG. 2 is the grapplerspoons 212, 214. The grappler spoons 212, 214 may be configured with themain nozzle 203 between them and be configured to move in a pincermotion toward the nozzle 203 by a robotic actuator 216 and a hinge 218arrangement. In some examples, the hinges in the hinge, pivot bracket218 may be spring-loaded in order to impart a force on the spoons 212,214 to bias them together, and thereby work against the force of thepincher bracket 216 when it is extended or retracted by the pincheractuator 204. In some example embodiments, the grappler spoons include acushion 220, 222. In some examples, the cushion 220, 222 may be made ofor include closed cell foam, neoprene, gel filled pads, liquid filledpads, open cell foam, layers of foam of different densities, a foambacking with a gel filled pad on top, and/or any combination of theabove or other material that may cushion a target 250 when the grapplerspoons 212, 214 pinch the target 250. In some examples, the materialcontacting the target 250 is no more than 20-30 durometer in hardness.

In some examples, a pneumatic trash cleaning air jet 224 may be mountedto the end of the grappler spoon 212, 214 in order to help clear debris.In such examples, air holes may be configured on the end lip of thespoons and face in various directions to direct air toward foliage. Insome examples, a line of holes may be configured on the end lip of eachgrappler spoon 212, 214.

In some examples, a picker head 202 may include two grappler spoons 212,214. In some examples, three spoons may be employed in a similar manneras those examples shown with two as in FIG. 2. In some examples, fourgrappler spoons may be configured in two axes around the picker head 202assembly. In some examples, alternatively or additionally, the grapplerspoons include a hinged and/or spring loaded portion at the end tobetter cushion the target 250 when pinched. In some examples, thegrappler spoons 212, 214 may pivot about the nozzle 203 to impart atwisting motion to snap a berry or other stem as discussed herein.

Robotic Arms with Picker Heads Examples

FIG. 3 shows two views, a side on 390 and top down 392 view of the samerobotic arm 360 and picker head assembly 302 picking targets 350 from aplant 380 on a planter bed 301. As discussed above, the picker headassembly 302 may include any number of vacuum grappling and/or pincherspoon features that may allow the picker head assembly 302 to grasp andpick a target 350. Any of various sensors may be employed as describedherein to locate and identify the targets 350, to create coordinates ofthe targets 350 to pass to the picker sub assembly 302 by computer, topick.

Seeker/Sensor Subassemblies

In some examples, the harvesting described herein is directed by aseeker subassembly that is able to identify targets for harvesting, passcoordinates for the targets to the picker subassembly for extraction.Such seeker subassemblies may include any number of cameras (visiblelight, thermal, UV or other), radars, lidars, lasers, acoustic locationfinders, GPS, inertial navigation systems, piezoelectric sensors, and/orany combination of these or other sensors to locate and identifytargets. The sensors may be in communication with a computing system,such as that described in FIG. 11 to send and receive data, commands,and/or any information to operate and share information regarding theirrespective sensor information gathered, including but not limited topixelated image data.

In some examples, a suite of these or other sensors could be placed atthe end of a robotic arm such as those shown in FIG. 4. In the example,a sensor 403 may be mounted to an articulating robotic arm 460 and isable to be maneuvered by a human operator and/or computer, to locate anddetect a target 450 in whichever manner the senor operates (light, heat,lidar, acoustic, radar, etc.). In such a way, the multiple degrees offreedom of the robotic arm 460 may be used to maneuver the sensor 403into line of sight 405 with a target 450. In some examples, multiplesensors 403 may be configured on a single arm 460. In some examples,multiple arms 460 may operate with their own or multiple sensors 403. Insome examples, sensors 403 may be mounted on a rotatable mount and/orrobotic arm 460, able to move and/or rotate in one, two, three, four,five, six, seven, or more degrees of freedom. In some examples, arobotic arm 460 includes sensors, picker heads, and/or multiple sensorsand/or picker heads, and/or a combination of sensors and/or picker headsas described herein.

In some examples, a frame portion (partially shown as 462) of theharvester and/or seeker subassembly may include mounted camera(s) 430configured to locate and detect targets 451. In some examples, one ormultiple cameras 432, 434 may be configured such that they have a lineof sight 442, 444 to the planter bed 401 and/or foliage 480 to locateand detect targets 451. In some examples, the sensors 432, 434 mayinclude rotatable mounts to swivel and/or rotate to view targets 451. Insome examples, as shown in FIG. 5, multiple sensors 432, 434, may beused in a stereoscopic arrangement to view the same field of view 442,444 from different angles, and thereby be used to create coordinates ofthe targets 451 by the computing system as described herein.

Camera Examples

In some example embodiments, the sensors described herein may include acamera and/or multiple cameras (for example, but not limited to, 403,432, 434, etc.) arranged so as to be able to view the target foliage andthereby the target agriculture to be harvested. In some examples,multiple cameras may be arranged on the seeker subassembly such thatdigital, pixelated images taken from the multiple stereo cameras may beprocessed by a computing system to create three-dimensional (3-D) imagesusing machine vision. In some examples, these images are made of pixelsand the computing systems is able to identify targets represented bypixels to be harvested and map the targets in three dimensions.

Examples of cameras which may be used in the described systems includestereo vision with resolution of 1920×1080 and frame rates of 30 persecond. In some examples, the cameras may include stereo vision with aresolution generally in the range of 1900×1000 and frame rates in therange of 10-60 per second. Other kinds of digital cameras may be used aswell, as these examples are not intended to be limiting.

In some examples, the cameras may be configured to acquiremulti-spectral or hyper-spectral imagery to enable the use of advancedanalysis algorithms for evaluating fruit health, quality and/or ripenessbased on the pixelated data. In some examples, the images gathered mayinclude those of a thermal imaging system for evaluating the temperatureof the berry to be harvested. These cameras may comprise of cooled oruncooled sensors generating area-scanned, point scanned, and/or linescanned images of at least 160×120 pixels. Some embodiments may utilizea single thermopile-based sensor to provide an integrated temperaturemeasurement of the mean temperature of the target berry to be used inanalyzing ripeness and/or target berry health.

In some examples, the target identification may be automated by thecamera systems. In some examples, the target identification may be aidedby a human who is analyzing a visual representation of the image datasent by the camera(s) wirelessly at a viewer/operator station asdescribed in FIG. 7, FIG. 9, etc.

Mapping and Passing Target Coordinates

In some example embodiments, additionally or alternatively, as describedherein, sensors onboard the harvesting systems such as machine visioncamera and computing systems may be used to map targets in threedimensions and pass the coordinates to the harvester subassembly forharvesting. In the remote GUI examples, as shown in FIG. 6 and FIG. 7,the user or computer may select targets from images for the computer tomap. These mapping coordinates may be described in a global coordinatesystem such as Universal Transverse Mercader (UTM), or a localcoordinate system frame relative to the coordinate system defined by thethree-dimensional imaging system on the harvester. In some examples, athree-dimensional X,Y,Z coordinate system may be employed using ananchor point in the camera view and/or on the traversing machine itself.In some embodiments the machine vision cameras may be calibratedextrinsically and intrinsically determine their location relative toother elements in the harvester and characterize the focal length,offsets, and lens distortions of the camera. In some embodiments, theintrinsic and extrinsic calibration parameters may be generatedautomatically by the system using on-harvester reference points;additionally or alternatively separate calibration targets with knowngeometries may be used.

The various sensors described herein including but not limited tovisible light cameras, infrared cameras, ultraviolet cameras, lidars,radars, lasers, or other sensors may be used to scan the produce plantsand identify targets. Using the automated, semi-automated, or manualselection processes and systems described herein, the systems couldgenerate coordinates for selected targets, including by the computerand/or human selection as in a GUI on FIG. 6 and/or FIG. 7. These mappedtarget coordinates may then be queued in a buffer or database, for theharvester subassembly to harvest in queued order, using the grapplersystems described herein. In some examples, after one target coordinatemay be added to the harvesting coordinate queue, more targets may beadded to the queue to be harvested in turn. In such examples, thetargeting subassembly, machine vision, and target mapping may occurwithout lag or delay in the handoff from targeting to harvesting, andnot be hampered by the limitations of the harvesting subassembly itself.

In such a way, in some examples additionally or alternatively, thetargeting subassembly may be mounted on a separate vehicle to travel atits own speed and send targeting mapped data to the harvestingsubassembly by wireless communications. In some examples, the targetingsubassembly may be a part of the overall harvesting machine and/orconnected to or in communication with the harvesting subassembly to passthe targeting mapped coordinate queue by wired communications to theharvesting subassembly. In some examples, a cloud or distributedcomputing resource may be utilized so that the targeting queue may berelayed or sent to the harvesting subassembly wirelessly as described inFIGS. 8, 9 and/or 10.

In some examples, the mapping may be done early or before a harvestermachine may come down a row. Additionally or alternatively, in someexamples, mapping may be done just before harvesting, on the samemachine in some examples to minimize the variables of the berries and/orfoliage moving. Any time between target mapping and harvesting may beutilized, depending on the circumstances of the harvest.

In some examples, mapping information may be stored in a remote server,cloud server, or distributed system, for the purpose of future analysis(post processing) of the imagery to evaluate the condition of the plantas described in FIGS. 8, 9 and/or 10. Post processing operations mayinclude an evaluation of the plant for disease, nutrient deficiency,unripe berry inventory, and/or other plant damage. In some embodiments,the resolution of the imagery may be fine enough to resolve and detectdefects, disease, and/or pests on a per-target scale. Data gathering andanalysis on all types of agricultural specifics may be accomplishedusing the suite of cameras and/or sensors on the systems describedherein. For example, outputs of post processing operations may beutilized to selectively address in-field issues at a plant-local scalethat may otherwise require broad remedies using traditional methods.Other outputs of post processing operations may generate statisticaldata related to observations and measurements that are made while theharvester is operating in the field that can be advantageous to thegrowers business efforts.

Some examples may include close in targeting systems and/or systems thatare capable of passing target coordinates not only from a targetacquisition system to a picker head for ultimate harvesting, but in someexamples, to pass coordinates from one set of sensors to another set ofsensors to thereby better acquire a target more precisely, using moregranular coordinates as described above.

Some examples may include stand-off cameras that are capable of handingoff control to close in cameras at a specific threshold of distance, orat a specific trigger. In such examples, the computer may lock ontotarget visually using a wide lens/wide field of view camera, generatecoordinates for the identified target, and then hand off the coordinatesof that target to a narrower lens/narrower field of view camera whichmay allow for more granular and exacting target acquisition and picking.In such examples, the narrower field of view camera may be able to moreaccurately pick out targets that are partially obstructed, laying in adifficult-to-see orientation, or be nearby other targets. In such a way,the more precise narrower field of view camera system may be able todiscern the target from the non-targets and update the coordinates fromthe wide field of view camera, with more precise coordinates to pass toa picker head assembly for harvesting.

In such examples, use of the on-arm camera to perform visual servo-ingcan serve the purposes of accounting for variations in the harvesterorientation due to uneven field conditions, forward progress of themachine down the row and errors introduced in the mechanical tolerancesof the encoders and motors of the mechanical arm. This may also allowthe use of a more inexpensive robotic arm to help control the cost ofthe system.

In some examples, the gathering of imagery may be decoupled fromharvesting operations if a finer temporal scale is required to observefield conditions. In some examples, a separate, dedicated ground rovermay be employed with the sensor package to autonomously orsemi-autonomously roam the fields and evaluate conditions.

Stereoscopic Camera Examples

In some example embodiments, additionally or alternatively, the seekersubassembly robotic arms may include at least one camera fixedly mountedas described herein. In some examples, the seeker subassembly roboticarms may include at least one light system as described herein. In someexample embodiments, a single robotic arm may include a multitude ofcameras and light systems. In some example embodiments, additionally oralternatively, the cameras and/or lights may be integrated into theharvesting robotic arms. For example, referring to FIG. 1A, in someexamples, the picker 102 on the end of the robotic arm 160 could includea camera and light system. To help map coordinates of targets in threedimensions, stereoscopic camera arrangements may be used with computeranalysis.

In some examples, as described, these stereoscopic cameras may bemounted to the harvesting system. In such examples, by offsettingmultiple cameras with generally the same aimpoint or field-of-view, athree-dimensional view of a target may be made from more than onedirect-on angle. FIG. 5 shows an example with stereoscopic cameraarrangements which may be used to determine coordinates of a target.Such examples may utilize pixelated image data to assign coordinates totargets represented by specific pixels. Such coordinates may be passedto the harvester picker heads for picking.

In the general stereoscopic example of FIG. 5, two digital cameras, aleft camera 510 and right camera 520 may be mounted on the systemsdescribed herein and utilized to determine three-dimensional coordinatesof an identified target 550. For example, dimensions include, an imageplane 590 is some f distance 592 from a line 524 between the cameras510, 520, angles Θ 512, 522 between the respective optical axes 514, 524of the respective left camera 510 and right camera 520. Using thesedimensions, the image plane 590 coordinates in X and Y may be determinedfor the target 550 for both the left camera 516 and right camera 526.Using these coordinates 516, 526, the images captured by the cameras510, 520 may be used to determine a combined three-dimensionalcoordinate for the target 550 and pass that coordinate to the harvestingsystem for picking as described herein, in some examples, in a queuewith other target coordinates.

In some examples, the stereo techniques include fully calibrated camerasto accurately determine the distance of targets in the images. In someexamples, the imagery from the left camera 516 and right camera 526 in astereoscopic pair is processed using software on a computer processor toextract three-dimensional information from the target scene and target550. Additionally or alternatively, three-dimensional data processingmay occur in a dedicated hardware processor such as a customApplication-Specific Integrated Circuit (ASIC), Field-Programmable GateArray (FPGA), Graphics Processing Unit (GPU) or Vision or VideoProcessing Unit (VPU).

In some example embodiments, additionally or alternatively, a cascade ofhierarchal cameras may be employed on the systems. In such examples,larger scope or angled cameras may be used to identify one or moretargets from a wide angle. In such examples, a first coordinate mappingmay be calculated using the wide angle lens cameras. In such systems,the back-end computers may receive the first coordinate or mappedinformation and use that to focus a second camera system on the selectedtargets for a more refined or granular targeting. The second set ofnarrower angle cameras may be configured to hone in on the targets thatthe wide angle system first mapped, and refine or detail a tighter setof coordinates for each target. This arrangement of passing from wideangle camera systems to a second set of narrower camera systems mayallow for a tight control loop for the picker assemblies.

In some examples, one or more laser sensors such as range finders may beconfigured on the systems to find, locate, and map targets. In someexamples, lasers may be employed to augment other camera assemblies. Insome examples, lasers may be used exclusively. In some examples, eachpicker head may include its own laser system to be used as a rangefinder, a color differentiator, target painter. and/or other sensor forthe final picking action at the target itself.

In some example embodiments, additionally or alternatively, the seekersubassembly and/or harvesting subassembly robotic arms may use sensorsto identify foliage blocking targets and use the sensor data to maneuvera foliage moving flipper or pneumatic air jets configured to alter,move, displace, and or otherwise gently maneuver the foliage of theplant to better expose the target berries for harvesting.

Automation and Remote Graphical User Interface (GUI) Examples

Additionally or alternatively, the systems described here may be used toharvest agricultural targets in an automated, semi-automated, or evenmanually controlled manner. In some examples, the semi-automated mannermay be arranged in a remote setting, allowing for a human to interactwith camera views from the harvester to help target the produce.

The variations on these options depend on how much a remote or localcomputing system may be programmed to identify and harvest a target. Forexample, in a fully manually controlled system, a human operator maycontrol the movements of both the seeker system and the harvestingsystem. In such examples, by remote control using a joystick or othercomputer driven operating device(s) a human could scan the rows ofplants for a target using the camera systems, and even maneuver therobotic arms that the camera systems are connected to, to identifytargets, and then use a control system such as a joystick to maneuverthe picker head assembly to the target, and then harvest the target asdescribed herein. Such examples would allow for remote operation of thesystems such as by wireless control to allow for human controllers to bestationed anywhere in the world, through combination of wired andwireless uplinks.

FIG. 6 shows an example GUI where a human may interact with atouchscreen or other interface to select targets using the computersystems. In some examples, this GUI is presented on a remote system, incommunication with the harvester machine. The screen may depict a cameraview or image captured by cameras from the harvester, and presentseither a still or moving video image in whichever energy spectrum thecamera operates in. For example, in a visible camera arrangement, thescreen may show what a visible light camera image captured, allowing ahuman operator to select whichever targets 650 she so chooses from theimage data. The software then boxes 687 the target on the screen aroundthe selected target 650. The computer systems on the harvester and thesensors, whichever they may be, then generate a coordinate for thatselected target, store and pass that coordinate as a viable target forharvesting to the harvesting subassembly. In some examples, a falsecolor, or color mapped image may be displayed to the remote operatorenhance the contrast or visibility of certain features of the target tobe picked. This may include features such as, but not limited tobruising, contamination and other undesirable deformities. Categories oftargets may be selected by the user in such a way, to indicate whichtargets are for harvesting, which for removal, and which to pass forlater harvesting as underripe.

In some examples, the three-dimensional image data processed by and sentfrom the camera(s) and/or other sensors may allow for a virtual realityenvironment to be created for a human user in a remote location. In suchexamples, a virtual reality headset or display may be utilized by auser, remote or close to the harvester, to locate and identify targetsusing the camera image data and thereby send the target mappingcoordinates to the harvesting machine for harvesting. For example, auser with a virtual reality headset may view camera data from thesensors on the harvesting system, and use a user input device such as atrigger, button, remote, and/or other indicator in conjunction with thevirtual reality display to mark and/or otherwise identify targets in theimage data. In some examples, remote users may utilize touch screentablet computers to view still images taken by sensor(s) at theharvesting assembly. Such users may select targets using the touchscreen and thereby identify targets to harvest for the system.

In some examples, data created by the cameras and data created by thehuman selection of agriculture may be stored by the computing device. Insuch examples, the identification data which in some examples may belabeled by a human operator using an interface, may be amassed andpost-processed. In such examples, after much data regarding targeting,identification, and/or harvesting is gathered, a neural network enginemay be trained and eventually may be able to replicate some or all ofthe human targeting using the labeled datasets.

FIG. 7 shows another example GUI where a human may interact with atouchscreen or other interface to select targets using the computersystems. In the example FIG. 7, a static image is displayed 710 capturedfrom the harvester camera sensor systems as described herein. In someexamples, the image may be a real-time moving live image. In someexamples, multiple, stereoscopic static images are displayed. In someexamples, virtual reality images may be displayed for a user to selectand classify targets. Any kind of image display may be utilized with thesystems and methods described herein for a remote and/or local humanoperator, live streamed and/or statically captured images.

In the example FIG. 7 shown, the image displays multiple targets 750,751, 752, 753, 754 around which an indicator shape 760, 761, 762, 763,764, has been rendered. Such an indicator may be rendered by thecomputer system, using algorithms and image processing in an attempt toautomate the target identification. In fully autonomous mode, thecomputer system would select all the targets without human intervention.In semi-autonomous mode, the computer system may select targets that itis able to identify, and presents them for editing to a human operator.In a fully manual mode, no targets are identified by a computer, and thehuman operator must select them all.

In some examples, the human operator, in semi-autonomous, or manualmode, may touch the screen where a target is located, in order toidentify the target for mapping and harvesting, causing the computer todisplay the indicator 760, 761, 762, 763, 764. In some examples, acursor may be manipulated by a human user using a joystick, mouse,keyboard, touchpad, augmented reality button, virtual reality trigger,or any other kind of human interface that allows selection of a targetfrom the screen. Once selected, the computer may utilize the pixellocations of the target to map coordinates of the selected target topass and/or queue to the harvesting system for harvesting.

In the example of FIG. 7, more than one kind of indicator 760 may beused by the computer or human operator. For example, the user interfaceshows different available icons for which a human and/or computer mayclassify a selected target. In the example, a good target 730 isindicated as one for harvesting. A trash target 732 is one indicated topick and discard in order to clear the foliage of bad targets. Anunderripe target 734 may indicate a target that should be left alone,not picked but potentially monitored for later harvesting. The userinterface may utilize many varieties of identifying to the user these orother classifications of targets. For example, a color system may beused to indicate good, trash, or underripe targets. In such a system, agood indicator 730 may be blue in color, a trash indicator 732 may bered in color, and an underripe indicator 734 may be yellow in color. Anycolor may be used in each of these indicators for the human operator tomore easily identify which of the classifications each target 750 is. Insome examples, alternatively or additionally, dashed lines may be usedfor the indicators 760, highlighted areas, lowlighted areas, blinkingindicators, or any other kind of visual identifier may be used alongwith or to replace colored indicators.

In some examples, the computer system may keep a tally 740 of theclassified or categorized targets on the screen. In some examples, abutton to request image review by a human supervisor may be presented742. In such examples, the image with classified targets may be sent toa supervisor user to run a quality control analysis on the selections inwhich case, the supervisor may make edits and/or send the image back tothe human operator for further editing.

In some examples, the user interface may include a button indicatingthat the human user is done with the image and target classification770. By selecting the done button 770, the human may then signal to thecomputer to send the coordinates of the targets for harvesting to theharvester system. In some examples, after a done selection is made,another image is presented for target selection. Alternatively oradditionally, a done and pause button 772 may be presented which wouldallow the human user to send the current image selection to the computersystem for harvesting but then pause the next image presentation forsome duration of time. In such examples, the computer may send the nextimage to another user for target selection, while the other user takes abreak or ends a shift. In some examples, the user may be given a timelimit to make the target selections and/or to submit target imagesbefore the next image is presented. In some examples, a time bar 774 maybe shown on the screen which changes color or diminishes in size at aspecific rate, indicating to the human user the time before the nexttarget image is presented. Such a time bar 774 may help human users tobudget their time and keep on task while the harvesting system isoperating in the field. This time discipline may be useful because inreal-time harvesting examples, the remote operator may not see theharvester moving down a row to harvest targets, but in another part ofthe world, the system may be moving and harvesting. Getting behind onselection of targets may hinder the progress of the harvester, and slowproduction. In some examples, a rate calculation 780 may be presented toindicate how many targets per hour, per minute, per day, or any otherrate are being harvested. Such a calculation may be used to aid asupervisor to speed up slow users, allow for rewards to be handed out tothe most productive workers, and utilize the data for optimization oftechniques and methods of harvesting in the overall remote system.

In some examples, an end session button 776 may allow a human user tostop the selections and end a session of image selection. In someexamples, the user interface of FIG. 7 may pre-fetch and cache one ormore frames for the human operator to evaluate. As maximum productivityis the desired performance level of the entire human-harvester system,pre-fetching frames can help reduce the wait-times that couldpotentially be associated with transmitting digital image files from theback-end computing system to a remote operator located anywhere in theworld. Pre-fetch queue depth and timing parameters need to be carefullyconsidered to achieve the desired minimum down-time performance of theoperator while maximizing the forward velocity a harvester in the fieldcan achieve. This is achieved by preventing harvesting pauses resultingfrom delays in receiving pick decisions from a remote operator.

The other extreme of control systems would be a fully automated system.In such a system, the traversing machines would move down a row ofagricultural targets and the seeker subassembly would use machinelearning/artificial intelligence/neural networks/and/or otherprogramming to seek out and identify targets with the seekersubassemblies and then harvest them as described using the picker heads.Such examples would depend on computer algorithms and programs todetermine using the inputs from the cameras and sensors, what a targetmay be and where they are located. For example, a color camera may beused by the computing system to detect a red strawberry amongst thegreen foliage of the plant it is growing on. Then a laser system and/orstereo camera vision could be used to determine an approximate locationand range from the system and the computers could use that informationto triangulate a three-dimensional coordinate system and identify wherethe target is located in space, relative to the traversing machine.Next, the coordinates could be passed to the harvesting subassemblywhere the picker heads may attach to and harvest the target strawberry,in some examples using its own sensors such as cameras and lasers.

The middle-ground option between the fully automated and the manuallycontrolled system would be some variant of semi-automated seeking andharvesting. The degree of semi-autonomy and which portions wereautomated and which manually controlled could vary from separatesubassemblies. For example, the seeker subassembly may be more manuallycontrolled with a human interacting with the cameras and sensors to helpidentify targets. In some examples, that may include a human interactingwith a graphical user interface “GUI” such as a touchscreen to identifya target displayed on the screen. FIG. 6 shows an example screenshot ofwhat a human interaction screen may look like. By tapping or boxing thetarget 687, a human could help identify a target for the system to thenmap the coordinates.

In such a system, the computer system may then determine and use theidentified target coordinates to pass to the harvesting subsystem forharvesting the targets. In some examples, machine learning or artificialintelligence may even be used to present potential targets 688 to thehuman interface screen for the human to either confirm 688 or deny 686by tapping or selecting. In some examples, the interface screen may thenindicate whether the target has been approved or not, in the examplewith a check mark or X mark. These GUI examples are merely exemplary andnot intended to be limiting. Correction data provided by the human toany selection that is inferred by the on-board neural-network can alsobe utilized as training data to be fed back into the development of theneural network.

In any of the above examples of automation, the sensors onboard theharvesting system may be used to create, track and pass coordinates ofthe targets for harvesting by the picker assemblies.

In some examples, a precision navigation system such as GPS or LIDAR maybe used to keep the planting assembly centered on the planting bed.Additionally, the location of individual plants placed in the ground maybe recorded in a global coordinate system for such business intelligencepurposes such as tracking inventory.

FIG. 7 shows another example user interface, where a remote operator maybe able to help the computing system identify targets for harvesting. Inthis example, still digitized images, taken by cameras on or near theharvesting assembly are presented on a computer screen. Such examplescreens may not be near the harvesting system itself, and instead be aremote user station with wireless communications connecting the remoteuser to the harvesting system. In such a way, the targets may beidentified by a remote user, not subjected to the elements, in alocation that is easy to maintain. The information may flow throughcommunication channels for the digitized images, as well as theidentification of targets as described herein, using networks, switches,routers, and other wired and wireless computerized communication andcontrol systems.

In some examples, the computer, using information programmed into itregarding size, shape, color, temperature, fluorescence, or any othercharacteristic, may analyze a digital image and identify targets withinthat image. In some examples, such an image, with previously identifiedby the computing system targets, may be presented to a user as a stillimage. As can be seen in FIG. 7, the previously identified targets arehighlighted in a color-coded system. Using the image and an interfacesuch as a touch screen, mouse, joystick, voice control, eye control,wearable three-dimensional controller with gyros, or any other kind ofcontroller alone or in combination, an operator may select new targetsthat were missed by the computer, delete targets selected by thecomputer because they do not meet the correct criteria for harvesting,or otherwise identify targets for the computer.

In such a way, each time a user either ignores a correctly identifiedtarget, adds a new target that was missed, and/or deletes a computerselected target, the computer is able to store and analyze that data forfuture use. In such a way, the computer is constantly fed new models totrain its algorithms on for future target acquisition.

Neural Networks and Training Models for Artificial Intelligence Examples

Systems and methods here may include use of neural networks, machinelearning, and/or artificial intelligence programming to help identifytargets as described in FIG. 7 and FIG. 9. In such a way, theprogramming may learn, by being fed examples and/or models of whattarget color, size, shape, position, or other characteristics areacceptable for harvesting, which to wait and not harvest yet, and whichare to be removed as garbage. In some examples, enough model trainingmay be utilized to change from semi-autonomous, where a computerattempts to identify and/or classify targets for human operators toreview and edit, to fully autonomous modes, with the computer systemutilizing its software to identify, select, and/or classify targets,just as humans would do as shown in FIG. 6 and FIG. 7. This may taketime, such as a few harvesting seasons for the neural network and/orartificial intelligence models to be fed by data in actual harvestingsettings and environments, but over time, the model updates may allowfor the computer to make better decisions as it is trained and retrainedwith new data. In such a way training data may be fed back into the AIthrough careful quality control of inputted data. The neural-network maybe developed for one variety of strawberry, or may be developed tohandle multiple varieties of targets.

In some examples, control of the picker head, once it reaches apredetermined off-set distance from the mapped coordinates of the targetto be picked, control of the movement of the robotic arm may be handedoff to an internal guidance system that may lock onto the targeted berryand fine tune any discrepancies in the logged coordinates that may occurfrom the forward movement of the harvesting platform. Such an internalguidance system may utilize a neural network inference and/or artificialintelligence in conjunction with accumulated data gathered onto to makedecisions and send associated commands to the robotic picker headassemblies. In some examples, the predetermined offset distance is 6inches from the target. In some examples, the predetermined offsetdistance is between 3 and 10 inches from the target. In some examples,the predetermined offset distance is between 2 and 15 inches from thetarget.

The end result in a full automated or mostly automated systems, would beto minimize the number of humans involved over time, with targetidentification.

Computerized Network Examples

In some examples, the harvesting systems may be in communication throughvarious wireless and/or wired networks, with computing back-end systems,other harvesters, other sensor suites, as described in FIGS. 8, 9, and10.

For example, in FIG. 8 an example networked system is shown which couldbe used in the systems and methods here. In FIG. 8, the computersystem(s) 802 onboard the harvesting system which is used to operate thesensing, coordinate generation, and/or harvesting system, includingprocess any images from the various sensors including cameras takingimages of the targets and plants. Such image data may include pixel dataof the captured target images. The computer(s) 802 could be any numberof kinds of computers such as those included in the sensors themselves,in the robotic assemblies, image processing and/or another computerarrangement in communication with the camera computer components mayinclude those examples are described in FIG. 11.

As shown in FIG. 8, the image data captured may be transmitted to aback-end computer system 830 and associated data storage 832 for savingand analysis. In some examples, this may include the remote operatorswho are interfacing with the harvesting systems, selecting targets,and/or overseeing maintenance of the systems. In some examples, thecommunication may be a wireless transmission 810 by a radio, cellular orWiFi transmission with associated routers and switches. In someexamples, the transmission may be through a wired connection 812. Insome examples, a combination of wireless and wired transmissions may beused to stream data between the back-end 830 and the harvesting systemincluding cameras, robotic pickers, etc.

In some examples, the transmission of data may include transmissionthrough a network such as the internet 820 to the remote operators,back-end server computers 830, and associated data storage 832. Once atthe back-end server computer servers 830 and associated data storage832, the pixelated image data may be acted upon by the remote operatorsto choose targets to harvest. In some examples, the data may be usefulto train the neural network, and/or artificial intelligence models as togood targets versus targets to pass up. In such examples, the image andtarget data may be stored, analyzed, used to train models, or any otherkind of image data analysis. In some examples, the storing, analyzing,and/or processing of image data may be accomplished at the computer 802which is involved in the original image capture. In some examples, thelocal computer 802 and a back-end computing system 830 may split thedata storing, modeling, analyzing, and/or processing. Back-end computerresources 830 may be more powerful, faster, or be able to handle moredata than may be otherwise available at the local computers 802 on theharvesting machines. In some examples, the networked computer resources830 may be spread across many multiple computer resources by a cloudinfrastructure. In some examples, the networked computer resources 830may be virtual machines in a cloud infrastructure.

In some examples, additionally or alternatively, data storage 840 may beutilized by accessing it over the network 820 in a distributed storagemethod. In some examples as described herein, remote human operators mayutilize computer interfaces 890 to make target selections and sendmapped coordinates back to the harvesting system 802 for harvesting. Insuch examples, back-end computer systems 892 and/or server computersystems may work with the user interface screens and selection inputs890 to send and receive data regarding the images, selections,categories of selections, and/or target coordinates with the harvestingsystems 802. In such a way, remote operators may view image data fromthe sensors on the harvesting system 802, make target selections andclassifications, and the computer systems may generate targetcoordinates to allow for the robotic assemblies to harvest the targetsaccordingly.

More detailed network examples are found in FIGS. 9 and 10.

Example Computing Device/Architecture

In example systems described herein, various computing components may beutilized to operate the systems. For example, a communication computingsystem may allow for remote operation of the machines, sensors may sendinformation to a computing system to help differentiate targets fromnon-targets, target location and mapping information may be calculated,stored, sent, and utilized between the seeker systems and harvestingsystems, steering and driving instructions may be calculated andutilized, machine learning/artificial intelligence/and/or neuralnetworks may be employed by computing systems to find and harvesttargets, and any of the other computing operations as described herein.

In some examples, alternatively or additionally, a WiFi system/cellularsystem/Bluetooth system, or any other communication system, with theappropriate antenna system and a processor and memory as describedherein, may be used on a subassembly. In some embodiments, alternativelyor additionally, the hardware may include a single integrated circuitcontaining a processor core, memory, and programmable input/outputperipherals. In some examples, the hardware may contain one or morespecialized processing centers for running the neural network inferenceprocessing in an accelerated fashion using devices such as graphicsprocessing units (GPU), vision processing unit (VPU), and fieldprogrammable gate arrays (FPGAs).

In some examples, various computing components may be used in the seekerand/or harvesting subassemblies, as well as the communication systems,control systems, and/or any other portion of the systems describedherein. In some examples, multiple computing devices may exist on theharvesting platform and perform discrete functions associated withharvesting operations. In such examples, each computing device may beinterconnected, or in communication with other computing centers withina networking system such as Ethernet or controller area network (CAN).

FIG. 9 shows an example computer architecture layout which may beutilized by the systems and methods described here, onboard theharvester. The example of FIG. 9 shows one harvesting system withmultiple picker arms and sensors. Each computer component in FIG. 9 mayrepresent the software utilized to effectuate the indicated featuresand/or the computer hardware to include a processor, memory, datastorage, and/or virtualized processors, memory, and/or data storage tooperate and carry out the instructions for the harvester, similar to, orthe same as, that shown in FIG. 11. The system controller 902 maycoordinate the subfunctions and be in communication with multiplecomputer components including but not necessarily limited to apropulsion system 904, and/or a steering system 906 to drive and controlthe movement of the overall harvesting system. Such a propulsion system904 may be in communication with motor(s) utilized in locomotion ormoving the overall system in the field. The steering 906 system may bein communication with a steering column, movable wheels, tank treads, orother maneuvering actuators utilized in turning the harvester in thefield.

The overall or master system controller 902 may be in communication witha navigation system 910 to receive and analyze positional data of theharvester, such as geographical and/or relative positional data within afield, which may include but may not be limited to, a Global PositioningSystem 912, Inertial Measurement Unit 914 for example a ring laser gyro,a magnetic gyro etc., Simultaneous Localization And Mapping system 916,and/or an Odometer 918. Each of the navigation systems 910 may includeall of the antennae, programming, chip sets, and/or other hardware andsoftware necessary to collect data and determine location, speed,distance, direction, or other navigation features necessary for theoperation of the harvesting system and transmit that navigation databack to the controller 902.

In some examples, additionally or alternatively, the cameras in theseeker/sensor subassembly and/or other cameras on the harvestingsubassembly may be used to identify and track an agricultural row downwhich the vehicle may be steered. In some examples, LIDAR and/or Radarmay be used to navigate the system in the field. The location sensingand/or steering may be fed into any computing system, either located onthe harvesting/seeking systems or remotely, in order to autonomously,semi-autonomously and/or allow for human activated remote steering. Anycombination of these or other systems may be used to locate and/or steerthe systems here.

In some examples, the harvester system may utilize self-steering, thatis computerized algorithms to send instructions to the propulsion 904and steering systems 906, when it is harvesting on a row and humanmanual drive to steer the system for unloading accumulated berrycontainers and reloading empty containers, then finally steering thesystem back onto a new row to be picked. The system may have the abilityto be converted to full autonomous mode for turnaround at the head landsas well as unloading and loading berry containers.

The controller 902 may also be in communication with a field network920. Such a network 920 may be located in the physical field ofoperation and include one or more data radios 922, 924 which maycommunicate with the harvester and controller 902 through an optionalcommunication interface 926 which may include all the necessaryantennae, data sending and receiving hardware and software necessary foroff-board communications. In such examples, data such as image and/ortarget coordinate data, may be sent from the system 926 to local radios922, 924, etc. and then off to an internet router 928 to communicatethrough the internet 960 or other network as described in FIG. 8. Byoff-loading the data to a local field network 920 then internet 928,remote harvesting target selections may be made using the image and/orcoordinate data determined by the harvester, sent to a remote user fortarget selection/classification, and the coordinate data and harvestinginstruction sent back to the harvester for harvesting.

System controller 902 may provide security and isolation to preventunauthorized users from reaching key internal systems. Such segmentationof communications may include encryption of data sent and received fromthe harvester including image data sent and received with a back-endsystem such as described in FIG. 8.

FIG. 9 also shows the system controller 902 in communication with anetwork switch 930. Such a switch may be in communication with one ormore picking segments 940, 950 and their associated own network switches942, 952 respectively. Although the example in FIG. 9 shows two pickingsegments 940, 950, any number could be located on a harvester system andcontrolled by the controller 902. FIG. 9 shows just two picking segmentsas an example, not intended to be limiting, and more picking segmentssuch as but not limited to one, two, three, four, five, six, seven oreight picking segments. Reference will be made to the two examplepicking segments and the similar or identical component parts found ineach, but are not intended to be limited to just two, and could be anynumber as indicated here.

Each picking segment 940, 950, etc. may include many multiple componentparts including a network switch 942, 952 for communication with thesystem controller 902 by way of the main network switch 930. Such anetwork aggregation device 930, such as an ethernet switch, caninterface all harvester segments to the central system controller on theplatform. This can alternatively be achieved through multiplecommunications internal to the system controller 902.

Each picking segment 940, 950, etc. may also include a targetacquisition or an identification processor 944, 954 and/or a motionprocessor 947, 957 in communication with the respective network switches942, 952. The identification processor 944, 954 may include artificialintelligence subcomponents and/or neural network programming used tomake determinations of target selection and/or coordinate mapping oftargets using the image data from the cameras as described below. Eachidentification processor 944, 954 may also be in communication with anidentification camera 945, 955 or two cameras 946, 956. Such cameras945, 955, 946, 956 may provide the pixelated image data taken of thetargets in order to process for target selection, target coordination,and/or classification of targets as described herein, in some cases byremote operator selection.

In systems using multiple cameras, such cameras may be arranged in astereoscopic manner to generate three-dimensional coordinate data oftargets in the field as described in FIGS. 5 and 4. The number ofcameras for each picking segment is not limited to two, and couldinclude wide angle/narrow focus cameras, stereoscopic cameras,thermosensitive cameras, laser rangefinders, and/or any other kind ofsensors that work alone or in combination. In such a way, the processingcenter 944, 954, may assign a three-dimension coordinate in a worldsystem to each pixel gathered by each of the cameras 945, 955, 946, 956.Such a processing center may perform neural network processing toidentify all candidate harvest targets in the acquired imagery andtransmit all processed results to the system controller 902.Additionally or alternatively, the results may be directly transferredto the motion processer 947, 957 described below, for fully orsemi-fully autonomous examples of the system.

In some examples, another processor, a motion processor 947, 957 mayalso be in communication with the network switches 942, 952 and alsooptionally be equipped with artificial intelligence programming in orderto help determine relative motion and positioning of the harvester,cameras, picker assemblies, and/or image data used for target analysis.Such motion processors 947, 957 may be able to determine coordinatesystems of the targets based on the images taken by the cameras 945,955, 946, 956, the position of the overall system with data from thenavigation system 910 and instruct the robotic arms 948, 958 toward atarget for harvesting. In some examples, a servo camera 949, 959 may beutilized to focus on a target even if the system on which the camera ismounted is moving relative to the target. In such examples, motors inthe servo camera 949 may utilize feedback of a selected target to lockonto the target and move to keep the target in a field of view. In someexamples, the robotic arm 948, 958 may be instructed toward a target forpicking, based on the coordinates created by the system and targetselected by a human and/or computer system. The motion processing center947, 957 can receive targets to harvest from the system controller 902or identification processor 944, 954, in fully autonomous examples. Themotion processor 947, 957 may use vehicle position information from thesystem controller 902 to resolve the relative position of the target tobe picked and computes a path for the robotic arm 948, 958 to reach thetarget. Such coordinates may be updated based on navigation system 910updates and/or servo camera 949, 959 target acquisition updates. In sucha way, after a commanding motion, continuous feedback from the servocamera may monitor the progress of the motion of the arm 948, 958towards the picking target and real-time neural network processingidentifies and tracks the harvest target. The system may then harvest atarget with the robotic arm(s) 948, 958 and then move to the next queuedtarget image coordinate data to harvest the next target.

In some examples, neural network processing can be accelerated in themotion processor 947, 957 through the use of dedicated hardware such asgraphics processing units, video processing units, and/orfield-programmable gate arrays. Results from the real-time neuralnetwork processing can be used by the motion processor 947, 957 tocorrect the target path of the robotic arm 948, 958 motion to compensatefor variable conditions such as (but not limited to) forward motion,changing vehicle attitude, inaccuracies in the robotic arm, and physicaldisturbances. Servo camera 949, 959 imagery can also be utilized toavoid obstacles in the path of the robotic arm 948, 958 motion such asleaves, rocks, dirt and other potential harvesting targets. Uponreaching the harvesting target, the motion processor 947, 957 maycommand the actuation of the gripper on the robotic arm 948, 958 toacquire the target and deposit the target for harvesting.

FIG. 10 shows a computer architecture diagram of the various computerand/or software components that may be in communication with one anotherto be used in the systems and methods described herein responsible fororchestrating harvesting operations. FIG. 10 is a computer architecturediagram expounding on an aspect of that shown in FIG. 9. As in FIG. 9,FIG. 10 shows a back-end architecture with a back-end server 1030 andassociated data storage 1032 communicate through an internet router 1034and through a network 1060 such as the Internet and/or a virtual privatenetwork with other components as described herein.

In some examples, images or other sensor data taken from the harvesters1090 in the field may be communicated through the field network 1020 anddata radios 1022, 1024 and an internet router 1028 and through thenetwork 1060 to the back-end server 1030 to be analyzed. Such analysismay include actively assigns frames to be evaluated to a pool of remoteusers 1040 and thereby their user interface 1042 through the network1060 and router 1044. In such examples, these remote users may be taskedwith indicating or labeling the provided image data with ahuman-evaluation of the correct harvest-targets in the image data asdescribed in FIG. 6 and FIG. 7 above.

The backend system 1030 may route coordinates for harvest targets fromremote users 1040 to the appropriate harvester 1090 in the field whenthe remote user evaluation has been completed. The backend schedulingsystem may include algorithms to maximize the productivity of harvestingoperations as a whole to keep harvesters serviced and remote operatorsfully engaged with imagery to be labeled. The backend system 1030 may ormay not store 1032 acquired imagery and operational data for furtheranalysis to improve situational knowledge of harvester operationsregarding business efficiency. The backend system 1030 may employ theuse of a database 1032 to organize, store and retrieve results duringand after harvesting operations.

The computing architecture can include remote interface computer(s) 1040for remote human operators to interact with harvesting operations. Insome examples, the remote interface computer contains a graphical userinterface (GUI) 1042 that presents a human operator with imagery ofareas of harvest acquired by the in-field system 1090 or seeker. Theremote operator GUI 1042 may provide tools that allows a human toidentify and classify potential harvest targets in the scene such asdescribed in FIG. 6 and FIG. 7. The GUI 1042 may provide classificationsto identify the target as under-ripe, ripe, over-ripe, diseased ordamaged. The GUI 1042 can be optimized to provide classifications tomeet the needs of the particular crop being harvested. The presentedimagery may additionally display preliminary classification resultsinferred by the on-harvester neural network to speed up humanidentification efforts as described in FIG. 6 and FIG. 7. The remoteoperator GUI 1042 can take user input from an operator using inputdevices such as keyboards, mice, touch screens, eye-tracking, joysticks,or custom-developed human interface devices. Classification changes thatthe human operators perform can additionally be recorded by the backendcomputer system 1030 and utilized as training information to refine theperformance of the harvester on-board neural networks.

The computing architecture can include a Picking Manager/OperationsManager (PM-OM) 1050 computer for the purposes of managing day-to-dayand seasonal harvesting operations related to the harvesting system, incommunication through an internet router 1054 with the network 1060 andthereby the other components. The PM-OM computer 1050 may provide a GUI1052 with selectable controls to allow management of the system by ahuman operator. Actions performed by the GUI 1052 may includefunctionality such as user administration, harvester administration,maintenance tracking, harvesting performance report generation, productanalysis, product tracking along with other information and controlsrelated to harvesting operations, and/or any other administrativefunction alone or in combination.

The computing architecture can include a Field Manager Computer 1002 toprovide in-field human resources access to information and controlsnecessary for harvester 1090 operations. In some examples, asemi-autonomous operation of the harvesting platform 1090, humans may berequired to actively maintain the system to perform functionalitynecessary for operations. These operations can include fuel resupply,consumable packaging material loading, finished product offloading,cleaning, end of row alignment and maintenance. Further developments tothe platform can be made to minimize the need for in-field operators1002 to include operations such as self-loading/unloading, auto-fieldnavigation/driving and intelligent cleaning systems. The Field Managercomputer 1002 may utilize a GUI 1004 to provide information and controlsto an operator for local control of the system. Information presentedmay include system health status, present location, performance metrics,error conditions, diagnostics and/or fuel level or any other logisticalinformation and control. Controls 1004 presented to an operator caninclude system enable, system stop, diagnostic controls, exteriorlighting, emergency stop and/or manual driving modes or any other formof navigation or other commands. The field manager computer 1002 mayinterface to a harvester 1090 through a wireless connection provided bythe in-field network 1020. Additionally or alternatively, the fieldmanager computer 1002 may contain a direct communications link to theharvester 1090 using interconnection technologies such as USB, Ethernet,serial, fiber, CAN, Bluetooth, NFC and/or private WiFi interfaces or anycombination of these or other communication methods.

FIG. 11 shows an example computing device 1100 that may be used inpracticing example embodiments described herein. Such computing device1100 may be the back-end server systems use to interface with thenetwork, receive and analyzed data, as well as generate test resultGUIs. Such computer 1100 may be a mobile device used to create and senddata, as well as receive and cause display of GUIs representing data. InFIG. 11, the computing device could be a smartphone, a laptop, tabletcomputer, server computer, or any other kind of computing device. Theexample shows a processor CPU 1110 which could be any number ofprocessors in communication via a bus 1112 or other communication with auser interface 1114. The user interface 1114 could include any number ofdisplay devices 1118 such as a screen. The user interface also includesan input such as a touchscreen, keyboard, mouse, pointer, buttons,joystick or other input devices. Also included is a network interface1120 which may be used to interface with any wireless or wired networkin order to transmit and receive data. Such an interface may allow for asmartphone, for example, to interface a cellular network and/or WiFinetwork and thereby the Internet. The example computing device 1100 alsoshows peripherals 1124 which could include any number of otheradditional features such as but not limited to cameras, sensors 1125,and/or antennae 1126 for communicating wirelessly such as over cellular,WiFi, NFC, Bluetooth, infrared, or any combination of these or otherwireless communications. The computing device 1100 also includes amemory 1122 which includes any number of operations executable by theprocessor 1110. The memory in FIG. 11 shows an operating system 1132,network communication module 1134, instructions for other tasks 1138 andapplications 1138 such as send/receive message data 1140 and/or SMS textmessage applications 1142. Also included in the example is for datastorage 1158. Such data storage may include data tables 1160,transaction logs 1162, user data 1164 and/or encryption data 1170. Thecomputing device 1100 also include one or more graphical processingunits (GPUs) for the purposes of accelerating in hardwarecomputationally intensive tasks such as execution and or evaluation ofthe neural network engine and enhanced image exploitation algorithmsoperating on the multi-modal imagery collected. The computing device1100 may also include one or more reconfigurable hardware elements suchas a field programmable gate array (FPGA) for the purposes of hardwareacceleration of computationally intensive tasks.

The computing architecture for the harvester can be described as adistributed computing system comprising of elements or processingcenters that exist on the harvester, a central server system which mayor may not be a cloud-based resource and an operator processing system.Each of these processing centers are interconnected through an IPnetwork which may include local private wireless networks, private widearea networks and/or public networks such as the Internet. Computationaltasks are divided such that real-time tasks are executed on the localharvester processor, post-processing operations and non-real timecomputation are executed on the central server and user-interfacecomputation are performed on the operator processing center.

Lighting Examples

In some example embodiments, the seeker subassembly includes variousspecialized lighting features which may be used to find and identifytargets. Such lights may be configured on the ends of robotic arms,integrated into robotic arms that include picker heads, mounted on theharvester assembly or sub-assemblies, and/or mounted on cameras.Examples are shown in FIGS. 1A and 1B. Such lights may be fixed ontoother sub-assemblies on the seeker assembly and/or harvestingsub-assembly and be in communication with a computer system to turn on,change intensity, change light source, switch wavelengths, etc.

In some examples, such specialized lighting may be configured to emit acertain wavelength or spectrum of wavelengths such as but not limited tovisible light, infra-red light, and/or ultra-violet light. In someexamples, the lighting may be at a wavelength that excites items tofluoresce. In some example embodiments, light spectrum filters may beused by the cameras described herein to filter out or delete wavelengths of light that would otherwise block out any fluorescentproperties reflected or emitted by targets such as berries.

In some examples, the specialized lighting may be comprised of lightemitting diodes (LEDs) which are tuned to emit light at a specificfrequency. In some examples, that frequency may be a combination of400-500 nm (blue) and 600-700 nm (red). In some examples, the lights maybe LED lights. In some examples, the lights may be incandescent lights.In some examples, the lights may be halogen lights, fluorescent lights,metal-halide, neon, high-intensity discharge lamps, or any permutationor combination of any of the above.

CONCLUSION

As disclosed herein, features consistent with the present inventions maybe implemented by computer-hardware, software and/or firmware. Forexample, the systems and methods disclosed herein may be embodied invarious forms including, for example, a data processor, such as acomputer that also includes a database, digital electronic circuitry,firmware, software, computer networks, servers, or in combinations ofthem. Further, while some of the disclosed implementations describespecific hardware components, systems and methods consistent with theinnovations herein may be implemented with any combination of hardware,software and/or firmware. Moreover, the above-noted features and otheraspects and principles of the innovations herein may be implemented invarious environments. Such environments and related applications may bespecially constructed for performing the various routines, processesand/or operations according to the invention or they may includecomputer or computing platform selectively activated or reconfigured bycode to provide the necessary functionality. The processes disclosedherein are not inherently related to any particular computer, network,architecture, environment, or other apparatus, and may be implemented bya suitable combination of hardware, software, and/or firmware. Forexample, various machines may be used with programs written inaccordance with teachings of the invention, or it may be more convenientto construct a specialized apparatus or system to perform the requiredmethods and techniques.

Aspects of the method and system described herein, such as the logic,may be implemented as functionality programmed into any of a variety ofcircuitry, including programmable logic devices (“PLDs”), such as fieldprogrammable gate arrays (“FPGAs”), programmable array logic (“PAL”)devices, electrically programmable logic and memory devices and standardcell-based devices, as well as application specific integrated circuits.Some other possibilities for implementing aspects include: memorydevices, microcontrollers with memory (such as 1PROM), embeddedmicroprocessors, Graphics Processing Units (GPUs), firmware, software,etc. Furthermore, aspects may be embodied in microprocessors havingsoftware-based circuit emulation, discrete logic (sequential andcombinatorial), custom devices, fuzzy (neural) logic, quantum devices,and hybrids of any of the above device types. The underlying devicetechnologies may be provided in a variety of component types, e.g.,metal-oxide semiconductor field-effect transistor (“MOSFET”)technologies like complementary metal-oxide semiconductor (“CMOS”),bipolar technologies like emitter-coupled logic (“ECL”), polymertechnologies (e.g., silicon-conjugated polymer and metal-conjugatedpolymer-metal structures), mixed analog and digital, and so on.

It should also be noted that the various logic and/or functionsdisclosed herein may be enabled using any number of combinations ofhardware, firmware, and/or as data and/or instructions embodied invarious machine-readable or computer-readable media, in terms of theirbehavioral, register transfer, logic component, and/or othercharacteristics. Computer-readable media in which such formatted dataand/or instructions may be embodied include, but are not limited to,non-volatile storage media in various forms (e.g., optical, magnetic orsemiconductor storage media) and carrier waves that may be used totransfer such formatted data and/or instructions through wireless,optical, or wired signaling media or any combination thereof. Examplesof transfers of such formatted data and/or instructions by carrier wavesinclude, but are not limited to, transfers (uploads, downloads, e-mail,etc.) over the Internet and/or other computer networks by one or moredata transfer protocols (e.g., HTTP, FTP, SMTP, and so on).

Unless the context clearly requires otherwise, throughout thedescription and the claims, the words “comprise,” “comprising,” and thelike are to be construed in an inclusive sense as opposed to anexclusive or exhaustive sense; that is to say, in a sense of “including,but not limited to.” Words using the singular or plural number alsoinclude the plural or singular number respectively. Additionally, thewords “herein,” “hereunder,” “above,” “below,” and words of similarimport refer to this application as a whole and not to any particularportions of this application. When the word “or” is used in reference toa list of two or more items, that word covers all of the followinginterpretations of the word: any of the items in the list, all of theitems in the list and any combination of the items in the list.

Although certain presently preferred implementations of the inventionhave been specifically described herein, it will be apparent to thoseskilled in the art to which the invention pertains that variations andmodifications of the various implementations shown and described hereinmay be made without departing from the spirit and scope of theinvention. Accordingly, it is intended that the invention be limitedonly to the extent required by the applicable rules of law.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the invention to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. Theembodiments were chosen and described in order to best explain theprinciples of the invention and its practical applications, to therebyenable others skilled in the art to best utilize the invention andvarious embodiments with various modifications as are suited to theparticular use contemplated.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the invention to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. Theembodiments were chosen and described in order to best explain theprinciples of the invention and its practical applications, to therebyenable others skilled in the art to best utilize the invention andvarious embodiments with various modifications as are suited to theparticular use contemplated. Etc.

What is claimed is:
 1. A harvesting vehicle system comprising: aharvesting vehicle frame with computing device includes at least oneprocessor and a memory including picking control systems, navigationcontrol systems, and communication control systems, a pickingsubcomponent including a robotic arm in communication with the computingdevice; wherein the robotic arm including a picker head assembly toharvest targets including a vacuum assembly with a compressor, hose, andpadded spoons configured to remove the target from a target stem;wherein the vehicle includes multiple cameras in communication with thecomputing device, wherein the cameras are configured to capture and sendimage data to the computing device; the computing device furtherconfigured to create three-dimensional maps of targets using themultiple camera image data; the computing device further configured todirect the robotic arm and picker head to a selected target to harvestusing the three-dimensional maps of targets, the picker head assemblyconfigured to attach the vacuum assembly and padded spoons to theselected target, and retract the selected target for harvesting.
 2. Thesystem of claim 1 wherein the computing device is configured to send theimage data to a back-end computing system over a network and receivetarget selection using the image data from the back-end computingsystem.
 3. The system of claim 2 wherein the received target selectionregarding the image data from the back-end computing system, includes aselection of a category of each selected target.
 4. The system of claim3 wherein the categories of each selected target include grade, spoiled,immature, and ready to pick.
 5. The system of claim 2 wherein theselected targets are selected by the back-end computing system, usingimbedded neuro network logic, trained from models of human selectedtargets classified as ready-to-pick, immature, or spoiled.
 6. The systemof claim 1 wherein the computing device is further configured to utilizeclose in sensors to direct the picker head to a selected target once thepicker head is within a predetermined distance from the target using thethree-dimensional map.
 7. The system of claim 1 wherein thecommunication system includes wireless communication devices incommunication with the computing device navigation control system, thewireless communication devices configured to send and receive dataregarding navigation to wireless antenna in communication with aback-end computing system.
 8. The system of claim 7 wherein thenavigation control systems includes at least one of Global PositioningSystem, Inertial Measurement systems, Simultaneous Localization AndMapping systems, and an Odometer.
 9. The system of claim 3 wherein theback-end computing system is further configured to cause display of aninterface for a user including the image data to allow touch screenselection of targets to be sent to the harvesting vehicle computingdevice for picking by the picker head.
 10. The system of claim 9 whereinthe coordinates of the selected target are sent to a queue buffer at theharvesting vehicle computing device for picking by the picker head inqueue order.
 11. The system of claim 9 wherein the display of the cameraimage data includes preselected targets, preselected by the back-endcomputing system, based on training of models of targets, wherein thedisplay interface allows users to affirm or change the preselectedtargets for harvesting.
 12. A method of harvesting agriculture,comprising: traversing a harvesting vehicle frame across a row ofagricultural plants wherein the harvesting vehicle includes a computingdevice with a processor and a memory, wherein the computing deviceincluding target acquisition control, picking control, wherein theharvesting vehicle including a picking subcomponent with a robotic armwith a picker head assembly, wherein the robotic arm in communicationwith the computing device, the picker head assembly including a vacuumassembly with a compressor, hose, and padded spoons; capturing andsending image data to the target acquisition control of the computingdevice, using multiple cameras on the harvesting vehicle; identifyingtargets in the agricultural plants, by the target acquisition control ofthe computing device, using the image data; creating three-dimensionalmaps of targets, by the computing device using the image data;directing, by the picker control of the computing device, the roboticarm and picker head to a selected target using the three-dimensionalmaps of targets; and harvesting, by the picker control of the computingdevice, the target with the picker head assembly by attaching the vacuumassembly and padded spoons to the mapped target, and retracting thetarget.
 13. The method of claim 12 wherein the traversing and navigationof the harvesting machine is controlled by navigation control in thecomputing device.
 14. The method of claim 12 further comprising sendingand receiving target acquisition and navigation data from acommunication control in the computing device with an off-board system.15. The method of claim 14 wherein the communication control includescommunicating using wireless communication devices by sending andreceiving data regarding navigation and camera image data by wirelessantenna in communication with a back-end computing system.
 16. Themethod of claim 12 wherein harvesting with the picker head assemblyincludes sending commands to a picker head actuator to pinch paddedspoons together to secure a target, the target being identified by thecomputing device target acquisition control.
 17. The method of claim 12wherein harvesting includes receiving data at the computing devicetarget acquisition control, from close in sensors on the harvestingmachine; directing the picker head to an identified target, by thecomputing device target acquisition control once the picker head iswithin a predetermined distance from the target, determined using thethree-dimensional maps.
 18. The method of claim 14 wherein the targetacquisition data from the off-board system includes target selectionwith selection of a category of each selected target.
 19. The method ofclaim 18 further comprising, causing display, with the off-board system,of a display interface of the camera image data for a user, and allowingtouch screen selection of targets to be sent to the harvesting vehiclecomputing device for picking by the picker head.